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Introduction and Acknowledgments 

The following notes and related handouts are intended to provide partici- 
pants with “hard-copy” materials covering some of the issues to be discussed in 
the seminar. The seminar is “exploratory” in nature and may produce more ques- 
tions than’ answers at this time. 

I have had the benefit of discussions and briefings with intelligence 
community members from CIA, DIA, and the Department of State as background prep- 
aration for this seminar. It is clear that the time committed has allowed for 
only a brief glimpse of a large and complex “system”, however, the time spent 
should aid in better focusing my concepts and tools of evaluation onto the 
concerns of the community. 

I thank those who have given their time and efforts to prepare me and hope 


the seminar will repay them in part. 


rer 
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1. Framework For Evaluation 


A. Concepts and Terms 


Analysis = future oriented - which alternative/option to choose 
and why? 


Evaluation = present to past orientation - what have we got? How 
have we done? 


Evaluation is concerned with three related, but distinct aspects that 
“build” upward in a hierarchal fashion: 
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How these aspects relate can be shown in a general "model" of any activity 
(field reporting for example) as a flow processing system that “produces” 
certain goods or services which are then used by end "consumers". The 
consumer use results in certain,impacts and/or end effects. 
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B. Quality 
Evaluation of Quality would concern itself with the Output(s): 


~ definition of output - what is the unit “produced” (goods, - 
service)? How will I know an output unit? 


- attributes of the output (implicit in the definition of the 
unit is the concept of “quality” ~- what perceivable attributes 
will be taken as making up an acceptable unit of output)? 


The attributes that represent “quality” of the output unit should be, but often 
are not, as specific, measurable, and quantifiable as possible. 


C. Efficiency 


Evaluation of Efficiency concerns itself with the relationship between the 
Output(s) and Input(s). These are the general categories of efficiency evalua- 
tion or assessment: ‘ 


.1. Work Measurement (WM) ~ relates the human resources (hours, 


Pe oa years) consymed” in the production of a unit of output. 


2. Unit Cost (UC) - relates the cost of all accountable and allocat— 
able resources to the production of a unit. 


3. Productivity Index (PI) - relates any “meaningful” measure of out- 
froductivity ineex Aris 4 
»y put to any measure of input. 


a 
Although these three forms of efficiency evaluation may appear in different ways 
there are only the three basic forms given above. 
e 
f 
D./. Effectiveness 


Evaluation of Effectiveness requires that the “results” and/or "effects" of 
the outputs be considered. There are, perhaps surprisingly, only two approaches 


that are available to evaluate effectiveness: 


Cost-effectiveness - the relationship of results to stated objectives. 


Benefit-cost - the relationship between the “value” of the results/effects 
obtained (benefits) and the "value" of the resources 
(inputs) consumed (costs). 
‘ * : 
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In cost~effectiveness analysis the stated objectives should be result related - 
not output orientated (often this is not true). Benefit-cost analysis requires 
the “valuation” (via shadow prices) of both resources (inputs) and the results/ 
effects “produced” by the outputs. There are two issues in the evaluation of 
effectiveness: 


- do your outputs “produce” a result/effect? 


- does your effect have "value"? 


Approach Effect? Value? 


Cost-Effective Yes 


Benefit~-Cost 


E. Evaluation - A Management System 


The three aspects of evaluation provide management with different items of 
information at different organizational levels. Different management levels 
within the intelligence community would require different evaluation informa~ 
tion. However, the provision of such information should be approached through 
the development of a hierarchically structured, integrated system where each 
level of evaluation utilizes the evaluation systems, reports, and basic data of 
the previous level. / 


Executive, policy,upper 


ffectiveness management. 


Efficiency 


Operating, line management. 
Quality 
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Ii. Evaluation and Quantification —- Methods, Tools, and Issues 
A. Objectives — De finition and Focus, 


Objectives must be capable of EOnETDLS measurement. 


Goal - General desired state or sondittons 2 (l-tele, Good Miho 
Not directly measurable, relatively timeless. 
Objective - Specific, tangible, measurable 
(time, amount, etc) result. 


Examples: 


Goal -— Promote general well-being and welfare 
of the public. 
Objective - Reduce heroine addiction in N.Y.C. 
area to or below the 1960 level. 


Goal - Stabilize Middle East political situation 

in relation to USA interests. 

Objective - Report at least 90% of all planned, 
anti-Shah demonstrations that involve 
over 1000 participants at least 12 hours 
before their initiation. 


There are only two classes of objectives: 
(a) Musts - These are absolute. They cannot be compromised; they 
must be totally achieved as stated. The objective sets a boundary, one 
side of which is acceptable and the other side of which is not. 
(b) Wants - These are relative. They can be compromised; they are 
not requirements and they may be traded off. The objective sets 
a range or region. One end of the region being the more "wanted". 


The Must and Want objectives may appear in two formats: 


(a) Upper-bound Must - 


‘\ 
Impossible \ 


\ Regio \ \Y WANT 
SNA SA 


Limit 


/ o> y 


Measure of Objective 
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(b) Lower-bound Must - 
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Limit Measure of Objective 


The term “objective(s)" is too broad and needs a focus when evaluation 
is being considered. Objectives can be related to Inputs, Outputs, 
Outputs/Inputs, or Results. 


Examples: 


Input - The agency will not expend more than $X on 
reporting related to XYZ in FY-79, 

Output - At least five reponts on XYZ will bé - 
submitted in 3rd quarter 1979. a 

Output/Input - The five reports on XYZ will be 
written using no more than 6 man-months 
of ABC staff time. 

Result - Reporting on XYZ will increase DEA intercepts 
of cocaine shipments into N.Y.C. port areas 
by 5% over FY-77,. 


Be Quality. 
The evaluation of quality requires that: 


1. The output units (goods and/or services) be identified 
and defined: “The manager of the decision unit should 
include in a statement of objectives information relating 
to both the services or products to be provided by the 
decision unit and the contribution of that output to 
achievement of the principal missions toward which the 
unit's efforts are-directed."” [OMB A-115, 5/5/78] 


2. Quality measures (attributes) be stated and defined - 
this is implicit in the definition of the output unit. 
The attributes, should be as quantifiable and objective 
as possible, however, subjective attributes will often be 
necessary or unavoidable. Note that the quality measures 
(attributes) have similar characteristics to objectives 
and, in fact, may be stated as objectives regarding the 
unit of output. : 
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Example - It is the agency objective(s) that all field reporting be: 
- timely 
- in response to requirements 
- reliable information 
~ valuable 
- of useful nature [Ref. DD Form 1480] 


Note -— To make the above attributes operational as measures of 
quality they would require: 
~ further definition or 
~ selection of (reference to) a “judge”. 

Taken together the set of attributes define the output’ unit. Any thing 
(good/service) which meets the minimal level of attributes is by definition 
a unit of output - if it does not, then it is not an output. 

The set of attributes used to evaluate quanaey consists, in most cases, 
of separate attributes of varying importance to the evaluator or decision 
maker. There are tools available to determine the relative importance of the 
attributes to the evaluator (paired weighting technique, rank correlation 
methods). 

The evaluation of quality is the composite of how an output item performs 
against the attribute and the importance of the attribute (the decision matrix 
technique would apply). 

C. Efficiency. 
The measurement of project or program efficiency is concerned with three 
basic areas: 


ny 


1. Work Measurement - the development of standard rates or times 
for the accomplishment of a defined work unit. The standard 
rate is defined as the amount of time (hours, minutes, seconds) 
required to produce a work unit. The work unit must be easily 
counted and verified, consistent over time, and similar in content 
throughout the organization. Knowledge of the standard rate and 
the total work units required per time period ( day, week, month) 
allows management to convert a particular work load into a state- 
ment of the human resources (people) required to produce the work 
load - justify staffing levels and allocations. 
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There are two approaches to the establishment of Work Measurement 


standard rates: 


fa) Engineered Techniques — 
Time Studies: 
Stopwatch 
Micromotion 
Synthesized Time Systems: 
Predetermined Standards 
Standard Data 
Work Sampling 


(b) Nonengineered Techniques - 

Job Tickets 

Historical: 
Time Logs ° 
Time Ladder 
Record Analysis (statistical) 

Estimating: 
Staffing Patterns 
Judgemental System 


The Nonengineered Techniques are most likely to be relevant to Field 
Reporting-Human Source Intelligence. 


2. Unit Cost Measurement - the relation of the output to the cost of 
resources consumed in producing that unit. The usual practice is to 
include all costs directly attributed to the production of the output 
unit plus an allocation of all other, indirect costs. Therefore, unit 
costs may include, in addition to direct labor (people) cost, the direct 

cost of supplies, travel, equipment, and facilities, plus a per unit 
‘allocation of indirect costs such as administration, overhead, 
research, development, and training. 


The costs may be expressed in any meaningful measure that management 
desires, but is generally a monetary unit ($). The monetary unit is often 
"adjusted" to account for inflation/deflation and expressed as constant-dollars 
of a base year. 


3. Productivity Index Measurement — the relationship. between a measure of 
output and one or more associated inputs - the units of output and 
input need not be common and usually are not. The Productivity Index 
may be based on any meaningful ratio of output measure (goods, 
services) to input measure management finds appropriate. 

The ratio may be based upon the amount of direct human inputs only 
(the inverse of the standard rate) or upon the costs of all. inputs 
(the inverse of the Unit Cost). 
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Each measure builds on the other. Work Measurement, the primary level of 
efficiency, deals with direct people resources (time) per unit of output. 

Unit Cost Measurement deals with all costs required to produce the output and, 
therefore, it is not limited to just personnel costs. However, Unit Cost 
Measurement builds on Work Measurement in that Work Measurement is used to sup- 
port the acceptability of direct people (personnel) costs. 

D. Relative Efficiency. 

The evaluation of efficiency can be undertaken from one, all, or any 
combination of the three viewpoints above - Work, Unit Cost, Productivity 
Index Measurement. The concept of “efficiency” is relative to both the 
viewpoint taken and the base chosen for comparison. The definitions for 
“efficiency” are: 

(1) Work Measurement: 
"Efficiency" = (Earned Hours)/(Actual Hours) 
Where: Earned Hours = Units Output x Standard Rate . 
(2) Unit Cost Measurement: 
"Efficiency" = (Standard Cost/Unit)/(Actual Cost/Unit) 
Where: 
Standard Cost/Unit = Standard Rate (hours/unit) 
x Standard Cost/Hour ($/hour) 
(3) Productivity Index: 
“Efficiency” = SoimuceyTapats 
Where: 
Outputs and inputs can be in any meaningful unit- 
usually not the same. Often stated as: 
(Units Output)/(Actual Cost Total) 


or 
(Units Output )/ (Actual Hours Total) 


_Z 
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Efficiency, when defined from a Work or Unit Cost Measurement viewpoint, will 
be a dimensionless ratio that could be less than, equal to, or greater than one 
(1.0). This ratio has meaning only relative to and in comparsion with the 
standard (earned hours or standard cost/unit) being employed. From the 
Productivity Index viewpoint, “efficiency” could be defined as a dimensioned 
ratio. This ratio can only be of use if there is a comparable ratio 

(previous time period, similar organization's operation, etc.). Standing alone 
any assessment of efficiency has no meaning. Only through comparison with 
standards, history, or other caisauapre organizations does a measure of 


efficiency tell management anything of value. 
E. First Definition of Effectiveness. 


Effectiveness is most commonly defined as the ratio of actual results 
(outcomes) to some stated end objectives or goals: 
Effectiveness = Results/Objectives : 
Common units of measure are required for both the results and the objectives in 
this ratio, but they need not be a monetary unit(s). The evaluation of 
effectiveness requires that specific, measurable program or project objectives 
be defined - a necessary, but often extremely difficult task. The stated objectives, 


from this view point, can be arbitrary and imposed from above. 


Example: 

Objective = Station XYZ will submit five (5) reports on topic ABC 
by 30 June 1979. 

Results = By 6/30/79 four (4) reports on ABC were written and sub- 
mitted by XYZ. 

Effectiveness = 4/5 = .80 (or 80%). 


a 


Approved For Release 2004/12/22 : CIA-RDP83M00171R001500160002-3 


Approved For @ease 2004/12/22 : CURD PSS MOONT MES 120018000259 


F. Second Definition of Effectiveness. 

The preceeding definition of effectiveness as the ratio of achieved results 
to some stated end objective or eon leaves much to be desired. There appears 
to be excessive room for arbitrary definition and the possible ommission of a 
reference ‘for evaluation. For example, agency management could have stated 
the objective as the submission, by Station XYZ, of only three (3) reports 
by 6/30/79. If there were an actual result of four (4) completed, then 
management could claim the following: 

Effectiveness = Achieved Results/Objective = 4/3 = 1.33 or 133% 
From management's viewpoint the reporting was deecctive, as defined, 
and if the data were available the reporting should, hopefully, prove efficient . 
upon comparison be eeeae activities. However, from a different viewpoint 
(governmental, national, general sabi te) how would the activities effectiveness 
be evaluated? How should it? What would be the “ground-rules"? Perhaps, in 
this example, no one uses the submitted reports. The four reports have ab 
measurable effects at all! Possibly, in the extreme, the only result of their creation 
is a measurable imposition of a need for their storage or disposal. | 

The flow-system concept of a project or program previously presented pro- 
vides an orientation for consideration of these questions. The reporting 
activity can be viewed as converting “social” resources (Inputs) into certain 
“social” returns (Results, Effects).* This view expands the first definition 
of effectiveness (Results/Objectives) to a consideration of the relation of 
the results (returns, effects) to the resources (inputs) required. 


[* Note - “Social” as used here refers to a general, aggregated, 
national, overall and public viewpoint ]. 
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Effectiveness now becomes defined as: 
Effectiveness = Results/Resources 

Although all social resources consumed by a program are not usually measured or 
accounted for (even in non-monetary terms), the monetary cost of resources pro- 
vides a base-line for evaluation. In fact, the resources required by an 
activity do not have to be measured in monetary terms, but they 46 tens to be 
measured (in some unit) for any evaluation or assessment of effectiveness to be 
possible. Since money is the usual common measure of resources, nearly all 
evaluations express the resource usage in monetary terms. Likewise, all the 
results or social returns from a program are not usually measured or accounted 
for, but for evaluation there must be at least some, although partial, measure- 
ment. There are two basic approaches to effectiveness assessment open to the 
evaluator — cost-effectiveness and benefit-cost analysis. The major distinction 
between the two is that cost-effectiveness analysis* primarily deals with situa- 
tions in which the input resources are apeecrsarin monetary terms and the 
results (for many bexdene) are not, while benefit-cost analysis addresses 
activities where both portions of the result/resource ratio are expressed in 
monetary units. The current level of effectiveness assessment in public sector 
projects and programs is mainly focused at cost-effectiveness analysis 
although some, but a small proportion, of programs are amenable to benefit-cost 
analysis. | 


[* Note - “Analysis” and “evaluation” are used as synonymous 
7 terms in this and following text ] 


G. Cost Effectiveness 
There are two basic approaches within cost-effectiveness analysis/evaluation: 
(1) fixed-cost, maximum effectiveness. 


(2) fixed-effectiveness, minimum cost. 
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In either case a relevant measure of effectiveness must be defined (casualty rate, 
response time (hrs), counter measure options, prisoners released) and the costs 

of resource inputs required by the different alternatives established. The 
definition of a relevant effectiveness measure is the critical requirement. 
Failure to do so can often lead to "a numbers game” which reflects little about 
the effectiveness of the activity. 

In any application of cost-effectiveness analysis, it is important to realize 
that the question “are the results worth the resources expended?” is implicitly 
answered in the affirmative. However, the causal link {T) between the project 
outputs and the results must be explicitly shown or there is no effectiveness. 

A primary concern and necessary outcome of a cost-effectiveness analysis is 
the establishment of this causal link. 
H. Establishing The CAs\jal Link. 
Some possible approaches to establishing that the outputs (reporting) has 7 
caused some result/effect are: 
- Observation 
- Content analysis of documents 
- Testing 
-- Records search 
- Interviews 
- Questionaires 
- Sociometric choices 
- Laboratory experimentation 
- Game playing 
- Physical examinations 
-. Physical evidence 
- Attitude tests 
- Opinion polls 
The mission here is to establish a “reasonable cause to believe (RCTB)” or 


probability that the outputs had an effect. It is unlikely (except in a few, 


special situations) that absolute proof of effect will be possible. For this 
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reason, the analysis/evaluation of effectiveness requires that all parties in- 
volved work from a “positive-constructive-competitive” point of view (as in 
scientific investigation). This requires that those with different and/or 
opposed positions respond by offering argumentation/support for their respective 
positions. The eanepes respond to the cases being built as the analysis/ 
evaluation warrants and not to “vested” or “self" interests. If this mode of 
operation can not be obtained, then there is little value to any evaluation 
effort. A similar "positive-constructive-competitive” environment for value 
(shadow price) determination in benefit-cost analysis/evaluation is discussed below. 
I. Benetit-Cost 

The question “are the results worth the resources expended?" can only be 
answered if the analysis/evaluation is able to: 


(1) establish a causal relation between program outputs and certain 
effects or results. : 


(2) derive economic measures for both the social costs of the : 
activity and the social benefits of the results -— from the public 3 
(taxpayer's) point of view. 

In this regard benefit-cost analysis is an approach, based upon a set of 
economic analysis concepts, that can aid in justifying the use and allocation of 
public resources to different programs. It is an approach that can aid top level 
management in justifying agency budgetary requests -'ranking of ZBB decision 
packages [OMB A-115]. The B/C approach must function from a "positive-construc— 
tive-competitive” viewpoint - as discussed above. The first step is to establish 
(RCTB) that the outputs “cause” certain effects. 

The second implementing step in benefit-cost analysis is the determination and 
“valuation” of the benefits and costs involved for the activities evaluated. 
There are economic and non-economic aspects to both the benefits and the costs 


when assessed from a public or social viewpoint. The costs are often easier to 
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establish as the resources expended ($, personnel) can be accounted for and pro- 
vide a base-line. This base-line may be adjusted (up or down) for social costs 
of a non-market derived nature since the market costs of the inputs may not 
truly reflect their social cost. The end result would be the “shadow price” 
of the Rate for each viable alternative being considered in the analysis. 

The benefits, being more in question, are “derived” or established in 
three major areas: 

- "Market" value - the price for the results, effects willingly 


incurred in "market" transactions by the public ~ charge-backs, 

fees-for-service, etc. . 

- Cost reductions/avoidances - the currently borne social costs/ 
expenditures that would be reduced or avoided by the results, 
effects - savings in current expenditure levels, lower error rates, 
fraud, etc. 


~ Indirect, intangibles - "values" such as “quality-of-life”, 
“independence”, “peace”, "feeling of security”, “cultural identity”, 
etc., which are agreed (by most) to exist, but are extremely 
difficult to quantify/monetize - this category includes 
legislatively mandated (Federal, State) goals or services that must 
be made available. . 
As in the case of the social costs of the alternatives, the end result 
of consideration of the above three areas ("market" value, cost/avoidance, 


intangibles) would be the derivation of the “shadow price” of the benefits 


for the results of the activities outputs. These shadow prices are an indica- 


tion of the “worth” of the various output results. They should not be taken as a 


definitive and absolute value measure, but rather a dynamic, working statement 
of the value seen in the alternatives considered. For this reason, benefit- 
cost analysis should and must be an overt and open process. The assumptions and 
analysis must be available for consideration and comment by as wide an audience 


as possible - the "positive-constructive-competitive” viewpoint. 
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